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Abstract.  The  temporal  logics  pCTL  and  pCTL*  have  been  proposed 
as  tools  for  the  formal  specification  and  verification  of  probabilistic  sys¬ 
tems:  as  they  can  express  quantitative  bounds  on  the  probability  of  sys¬ 
tem  evolutions,  they  can  be  used  to  specify  system  properties  such  as 
reliability  and  performance.  In  this  paper,  we  present  model-checking 
algorithms  for  extensions  of  pCTL  and  pCTL*  to  systems  in  which 
the  probabilistic  behavior  coexists  with  nondeterminism,  and  show  that 
these  algorithms  have  polynomial-time  complexity  in  the  size  of  the  sys¬ 
tem.  This  provides  a  practical  tool  for  reasoning  on  the  reliability  and 
performance  of  parallel  systems. 


1  Introduction 

Temporal  logic  has  been  successfully  used  to  specify  the  behavior  of  concur¬ 
rent  and  reactive  systems.  These  systems  are  usually  modeled  as  nondetermin¬ 
istic  processes:  at  any  moment  in  time,  more  than  one  future  evolution  may  be 
possible,  but  a  probabilistic  characterization  of  their  likelihood  is  normally  not 
attempted.  While  many  important  system  properties  can  be  studied  in  this  set¬ 
ting,  others,  such  as  reliability  and  performance,  require  instead  a  probabilistic 
characterization  of  the  system. 

The  first  applications  of  temporal  logic  to  probabilistic  systems  consisted 
in  studying  which  temporal  logic  properties  are  satisfied  with  probability  1  by 
systems  modeled  either  as  finite  Markov  chains  [14,  18,  12,  1,  20]  or  as  augmented 
Markov  models  exhibiting  both  nondeterministic  and  probabilistic  behavior  [22, 
19,  5,  20], 

Subsequently,  [10,  2]  considered  systems  modeled  by  discrete  Markov  chains, 
and  introduced  the  logics  pCTL  and  pCTL*,  that  can  express  quantitative 
bounds  on  the  probability  of  system  evolutions.  These  logics  can  thus  be  used 
to  reason  on  the  reliability  and  performance  of  systems.  They  are  obtained  by 
adding  to  the  branching  time  logics  CTL  and  CTL*  a  probabilistic  operator  IP, 
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such  that  the  formula  P>a<)>  is  true  at  a  given  point  of  the  system  evolution  if, 
starting  from  that  point,  the  probability  that  a  future  evolution  satisfies  <j)  is  at 
least  a. 

The  model-checking  algorithms  presented  in  [10,  2]  can  be  used  to  determine 
the  validity  of  pCTL  and  pCTL*  formulas  on  systems  modeled  by  finite  Markov 
chains.  Moreover,  [2]  considers  generalized  Markov  processes,  representing  fami¬ 
lies  of  Markov  chains,  and  shows  that  the  decision  problem  for  pCTL*  formulas 
on  generalized  Markov  processes  is  decidable  using  results  from  the  theory  of 
real  closed  fields.  However,  no  efficient  computational  method  is  given  for  this 
latter  problem. 

In  this  paper,  we  extend  the  logics  pCTL  and  pCTL*  to  systems  in  which 
nondeterministic  and  probabilistic  behavior  coexist.  We  model  these  systems  by 
probabilistic-nondeterministic  systems,  similar  to  the  augmented  Markov  models 
of  [19,  20].  Due  to  the  presence  of  nondeterminism  it  is  not  possible,  in  general, 
to  talk  about  the  probability  with  which  a  formula  is  satisfied,  but  only  about 
the  lower  and  upper  bounds  of  such  probability.  Therefore,  according  to  our  def¬ 
inition,  the  formula  TP>a(j)  (resp.  P <<,</>)  is  true  at  a  given  point  of  the  system 
evolution  if  a  system  evolution  starting  from  that  point  satisfies  <f>  with  a  proba¬ 
bility  bounded  from  below  (resp.  above)  by  a.  We  then  present  model-checking 
algorithms  that  verify  whether  a  system  satisfies  a  specification  written  in  pCTL 
or  pCTL*  in  polynomial  time  in  the  size  of  the  description  of  the  system. 

The  logics  pCTL  and  pCTL*,  together  with  these  model-checking  algorithms, 
provide  a  practical  tool  for  the  formal  specification  and  verification  of  the  perfor¬ 
mance  and  reliability  of  parallel  systems.  Nondeterminism,  as  already  recognized 
by  [22,  19,  5,  20],  is  in  fact  the  key  to  the  natural  modeling  of  parallel  probabilis¬ 
tic  systems  by  interleaving,  as  it  allows  us  to  model  the  choice  of  which  system 
in  the  parallel  composition  takes  a  transition.  Nondeterminism  also  gives  the 
flexibility  of  leaving  some  transition  probabilities  unspecified.  This  leads  to  sim¬ 
pler  system  models,  and  it  is  necessary  when  some  transition  probabilities  are 
unknown.  Leaving  some  transition  probabilities  unspecified  can  also  be  useful 
when  it  is  not  desirable  that  a  correctness  proof  of  the  system  with  respect  to 
some  specification  depends  on  the  value  of  those  probabilities. 

2  Probabilistic-Nondeterministic  Systems 

Following  an  approach  similar  to  [19,  20],  we  use  Probabilistic- Nondeterministic 
Systems  (PNS)  to  model  systems  in  which  probabilistic  and  nondeterministic 
components  of  the  behavior  coexist.  To  give  a  formal  definition  of  PNS,  we  first 
introduce  next-state  probability  distributions. 

Definitionl  (next-state  probability  distribution).  If  S  is  the  state  space 
of  a  system,  a  next-state  probability  distribution  is  a  function  p  :  S  <  ->  [0,  1]  such 
that  P(s)  =  1-  F°r  s£S,  p(s)  represents  the  probability  of  making  a  direct 

transition  to  s  from  the  current  state.  □ 


A  PNS  can  then  be  defined  as  follows. 


Definition 2  (PNS).  A  PNS  is  a  quadruple  17  =  ( S ,  s;n,  V,  r),  where: 


1.  S'  is  the  denumerable  or  finite  state  space  of  the  system; 

2.  s;n  6  S  is  the  initial  state; 

3.  V  is  a  labeling  function  that  associates  with  each  s  6  S  the  set  V^s)  C  V  of 
propositional  variables  that  are  true  in  s; 

4.  r  is  a  function  that  associates  with  each  s  6  S  the  set  r(s)  =  {pf , . . .  ,  pf  } 
of  next-state  probability  distributions  from  s.  We  denote  |r(s)|  by  ks.  □ 

The  successor  of  a  state  s  6  S  is  chosen  according  to  a  two-phase  process: 
first,  a  next-state  probability  distribution  pf  6  r(s)  is  selected  nondeterministi- 
cally  among  pt .... .  pf  ;  second,  a  successor  state  t  6  S  is  chosen  according  to 
the  probability  distribution  pf  on  S. 

This  model,  based  on  the  one  proposed  in  [19],  generalizes  the  approach 
of  [22]  by  allowing  a  simpler  encoding  of  the  parallel  composition  of  systems. 
To  see  how  parallelism  can  be  modeled  by  a  PNS,  consider  as  an  example  the 
parallel  composition  of  m  Markov  chains  Ai, . . . ,  Am.  In  a  PNS  II  representing 
Ai  ||  A'j  ||  ...  ||  Am ,  we  can  associate  with  each  state  s  6  S  the  next-state 
distributions  r(s)  =  {pf, . .  .,psm},  where  the  distribution  pf  arises  from  a  move 
taken  by  the  chain  A,-,  I  <  i  <  m.  In  this  way,  the  probabilistic  information  on 
the  behavior  of  each  chain  is  preserved  in  17,  and  the  choice  of  the  Markov  chain 
that  takes  a  transition  is  nondeterministic. 

We  define  a  reachability  relation  p  C  S  x  S  by 

p  =  {(s,t)  |  3ps  G  r(s)  .ps(t)  >  0}  . 

Then,  we  associate  with  each  state  s  6  S  the  set 

I2S  =  {s0siS2  . . .  |  .s-  •  -  .So  A  V//  fc  l\  .  p(sn ,  s„+i)} 

of  legal  infinite  sequences  of  states  beginning  at  s.  The  set  of  computations  of  a 
system  77  is  thus  I2Sin.  For  ui  G  I2S,  we  denote  with  ui\n  the  n-th  state  ofw,  with 
w|0  =  s. 

Moreover,  we  let  Bs  C  2^”  be  the  smallest  algebra  of  subsets  of  I2S  that 
contains  all  the  basic  cylinder  sets  {u  6  |  w|0  =  s0  A  . . .  A  w|„  =  sn}  for  all 

n  >  0,  s0,  .  .  . ,  sn  £  S,  and  that  is  closed  under  complement  and  countable  unions 
and  intersections.  This  algebra  is  called  the  Borel  cr-algebra  of  basic  cylinder  sets, 
and  its  elements  are  the  measurable  sets  of  sequences,  to  which  it  will  be  possible 
to  assign  a  probability  [13]. 


Minimal  and  Maximal  Probabilities 

Due  to  the  presence  of  nondeterminism,  we  cannot  define  a  probability  measure 
on  the  Borel  er-algebra  Bs.  However,  for  each  set  of  sequences  A  £  Bs,  we  can 
define  its  maximal  probability  pf  (A)  and  its  minimal  probability  pl(A).  Intu¬ 
itively,  pf  (A )  (resp.  pj  (A))  represents  the  probability  that  the  system  follows 
a  sequence  in  A  provided  that  the  nondeterministic  choices  are  as  favorable 


(resp.  unfavorable)  as  possible.  To  formalize  the  idea  of  favorable  and  unfavor¬ 
able  choices,  we  introduce  the  concept  of  strategies  (similar  to  the  schedules  of 
[22,  19,  5,  20]),  that  determine  which  next-state  probability  distribution  is  chosen 
for  each  state. 

If  the  system  reaches  the  root  s  of  C2S  following  the  sequence  s;nsi  . . .  s„s, 
we  can  assume  that  a  strategy  does  not  depend  on  the  “past”  sequence  ujp  = 
s;nS!  . .  ,sn.  In  fact,  we  are  interested  in  a  strategy  that  maximizes  or  minimizes 
the  probability  that  the  system,  starting  from  s,  follows  a  sequence  in  A:  as 
neither  A  nor  the  next-state  distributions  depend  on  ujp ,  such  strategy  also  need 
not  depend  on  ujp.  Formally,  a  strategy  is  defined  as  follows. 

Definition3  (strategy).  A  strategy  rj  is  a  set  of  conditional  probabilities  (i  \ 

s0si  . .  ,sn)  such  that  Ya=i  Qr,(*  I  so«i  •  •  -sn)  =  1,  for  all  n  £  IN,  s0,si,  ■  ■  ■ ,  sn  £ 
S,  and  1  <  i  <  kSn  .  □ 

When  a  system  behaves  according  to  a  strategy  r)  in  the  evolution  from 
s o  £  S,  and  has  reached  sn  following  the  sequence  sl: . ■ ■  s„ .  it  will  choose  the 
next-state  distribution  p*”  with  probability  Qv(i  |  so^i  •  •  .s„).  The  probability 
Pr^ ( t  |  so  ..  ,sn)  that  a  direct  transition  to  t  is  taken  next  is  thus  equal  to 

5Zi=l  Qv  (®  I  S0S1  •  •  •  sn  )Pj  "  (t)  ■ 

Therefore,  we  can  associate  with  each  finite  sequence  Sq  . .  ,sn  starting  at  the 
root  s  =  so  of  I2S  the  probability  ]]]["_ro1  Pr^(sJ+i  |  sq  ■  ■  ■ s; ).  These  probabilities 
for  the  finite  sequences  give  rise  to  a  unique  probability  measure  fis  v  on  Bs  that 
associates  with  each  A  £  Bs  its  probability  fi„tV(A)  [13].  We  can  then  define 
minimal  and  maximal  probabilities  as  follows. 

Definition4  (minimal  and  maximal  probability).  The  minimal  and  max¬ 
imal  probabilities  p~(A),  pf(A)  of  a  set  of  sequences  A  £  Bs  are  defined  by 

P7(A)  =  irfps,n{A)  =  sup/JSiI)(A)  □ 

V  7] 

Thus,  fi~  (A)  and  /u+(Z\)  represent  the  probability  with  which  the  system 
follows  an  evolution  .s\s:  as  . . .  £  A  when  the  nondeterministic  choices  are  as 
unfavorable  or  as  favorable  as  possible,  respectively.  In  general,  p+  and  p~  are 
not  additive  on  Bs,  as  the  following  lemma  states. 

Lemma  5.  If  At,  Ai  £  Bs,  with  Ai  n  A  2  =  0,  then 

Us  (^1  U  A2)  >  ps  (2\i)  +  fis  (A2)  pf  {Ai  U  A2)  <  pf  (Ax)  +  pf  (A2) 

and  equality  does  not  hold  in  general. 

The  minimal  and  maximal  probability  are  related  by  the  following  lemma. 
Lemma  6.  For  A  £  Bs,  it  is  p~  (A)  =  1  —  pf(I2s  —  A). 

Proof.  From  pS)V(A)  =  1  -  pStV(I2s  -  A),  we  have  pf  (A)  =  inf  vps>v(A)  = 
inf,,  (1  -  Ps,v{fis  ~  A))  =  1  -  sup^  pSi7l(f2s  -  A)  =  1  -  p+  (I2S  -  A).  □ 


3  Probabilistic  Temporal  Logic 


Syntax.  The  logics  pCTL  and  pCTL*  are  derived  from  the  branching-time 
logics  CTL  and  CTL*  [6]  by  introducing  a  probabilistic  operator  P,  with  the 
intuitive  reading  that  P ya<j>  (resp.  P<a(/>)  means  that  the  probability  of  holding 
in  the  future  evolution  of  the  system  is  at  least  (resp.  at  most)  a  [10,  11,  9,  2]. 
Formally,  we  distinguish  two  classes  of  formulas:  the  class  Stat  of  state  formulas 
(whose  truth- value  is  evaluated  on  the  states),  and  the  class  Seq  of  sequence 
formulas  (whose  truth-value  is  evaluated  on  infinite  sequences  of  states).  For 


pCTL*,  the  classes  Stat  and  Seq  are  defined  as  follows: 

V  C  Stat  (1) 

(j),  if>  £  Stat  =>•  <j>  A  ij),  -i</>  £  Stat  (2) 

£  Seq  =>•  A  (j),  E  (j),  P^c/i  £  Stat  (3) 

<f>  £  Stat  =>■  <f>  £  Seq  (4) 

(p,  r[>  £  Seq  =>•  <f>  A  if),  ~i<f>,  U<f>,  0<f> ,  <fi  U  ijj  (z  Seq  .  (5) 


In  the  above  definition,  IX  stands  for  one  of  <,  <,  >,  >,  and  a  £  [0,1].  The  logic 
pCTL  is  a  restricted  version  of  pCTL*,  and  its  definition  can  be  obtained  by 
replacing  the  clauses  (4),  (5)  in  the  above  definition  with  the  single  clause 

<f,  if)  £  Stat  =>•  □<)>,  O <j>,  <f>M  £  Seq  .  (6) 


Semantics.  For  a  formula  (j)  £  Stat,  we  indicate  with  s  |=  (j)  its  satisfaction 
on  state  s  £  S,  and  for  (j)  £  Seq  we  indicate  with  w  |=  cp  its  satisfaction  on  the 
infinite  state  sequence  u.  The  semantics  of  the  logical  connectives  and  of  the 
temporal  operators  is  defined  in  the  usual  way;  the  semantics  of  A,  E  and  P  are 
defined  as  follows: 


s  |=  A  <p  iff  Vw  £  12 s  .  uj 

(7) 

s  |=  E <p  iff  3ui  £  f2s  .  uj 

1=  4 

(8) 

s  |=  P>a<^  iff  jus“({tc  £  I2S 

1  w  1 =</>})>  a 

(9) 

s  |=  P <a<t>  iff  nf{{w  £ 

|  lo  |=  <()})<  a.  . 

(10) 

The  semantics  ofs  |=  P >„</i,  s  |=  P<a<(>  are  defined  in  a  similar  way.  This 
definition  has  a  very  intuitive  reading:  if  s  |=  P ><,</>,  it  means  that  regardless 
of  the  choices  made  in  nondeterministic  states,  the  probability  that  the  future 
evolution  satisfies  (j)  is  at  least  a  (and  similarly  for  s  |=  P<a(^). 

To  see  that  the  semantics  is  well-defined,  it  is  possible  to  show  by  induction 
on  the  structure  of  (j)  that  {uj  £  f2s  \  ui  |=  (j)}  £  Bs  for  every  (j)  £  Seq  [22],  We  say 
that  a  formula  (j>  £  Stat  is  satisfied  by  a  PNS  II ,  written  II  |=  <6,  if  s;n  |=  (j). 


4  Model  Checking 


We  now  present  algorithms  to  decide  whether  a  PNS  II  with  finite  state  space  S 
satisfies  a  specification  <p  written  in  pCTL  or  pCTL*.  We  will  prove  that  these 
algorithms  have  polynomial  time  complexity  in  the  size  of  the  description  of  II . 
We  first  give  the  algorithm  for  pCTL,  and  then  we  examine  the  one  for  pCTL*. 

The  algorithms  share  the  same  basic  structure  of  those  proposed  in  [8,  7]  for 
CTL  and  CTL*.  Given  a  formula  <p  E  Stat,  they  recursively  evaluate  the  truth- 
values  of  the  state  subformulas  ip  E  Stat  of  <p  at  all  states  s  E  S.  starting  from 
the  propositional  formulas  of  <p  and  following  the  recursive  definitions  (l)-(3)  of 
state  formulas,  until  the  truth-value  of  <p  itself  can  be  computed  at  all  seI, 

In  fact,  since  pCTL  and  pCTL*  differ  from  CTL  and  CTL*  only  for  the 
presence  of  the  P  operator,  we  can  use  the  same  techniques  proposed  for  CTL 
and  CTL*  to  deal  with  the  operators  A,  -i,  A,  E.  In  the  algorithms  below, 
therefore,  we  need  to  examine  only  the  case  corresponding  to  P. 

4.1  pCTL  Formulas 

Let  Pr+  <p  L=f  pf{{w  6  I2S  |  w  <P})>  Pr7  4>  =f  AU  ({w  G  I2S  \  w  |=  </>}).  From 
(9),  (10)  we  see  that  in  order  to  check  whether  s  |=  PM„<(  it  suffices  to  compute 
Pr+  ip,  Pr)-  <p.  Using  nip  -"0 ~'ip,  true U  ip,  and  the  relations 

Pr+  -nip  =  1  -  pr;  <p  Pr“  ~^<p  =  1  -  Pr+  <p  , 

derived  from  Lemma  6,  we  need  only  to  consider  the  case  of  <p  =  7  7/  ip.  Let  S&  = 
{s  E  S  |  s  |=  ^1}  be  the  set  of  “destination”  states,  and  let  Sp  =  {s  G  S  \  s  |=  7} 
be  the  set  of  “intermediate”  states. 


Computation  of  PrJ"  <f>.  It  is  useful  to  determine,  first  of  all,  for  which  states 
s  G  S  is  Pr“  ip  >  0.  To  this  end,  let  the  monotone  set  function  A  :  ‘2s  t->-  2s  be 
such  that,  for  ACS, 

A{A)  =  A  U  {s  G  IS,  |  Vi  G  {1, . . . ,  ks]  .  3t  .  (t  E  A  ApJ(t)  >  0)  }  . 

As  S  is  finite,  the  fixpoint  A°°(j4)  =  computable  in  at  most  l^l 

iterations.  Let  S> 0  -  The  following  lemma  states  that  this  is  exactly 

the  set  of  states  from  which  <p  can  be  true  with  probability  greater  than  0. 

Lemma  7.  s  G  S  —  S>  0  implies  Pr“  <p  =  0,  s  E  S^o  implies  Pr“  <p  >  0,  sG  Sd 
implies  Pr“  <p  =  1. 


We  still  have  to  determine  the  value  of  Pr“  <p  for  the  states  in  S'p  =f  A>(:  S'd. 
Each  s  G  Sp  will  choose  the  next-state  distribution  p?  :  1  <  *  <  ks  that  minimizes 
the  probability  of  getting  to  S Thus,  for  all  s  G  S'p  we  have: 

Pr7<i)  =  min 

1 <i<k. 


Pi  ^)Pr«  ^  Pi  (*) 


LirC/ 


(11) 


We  can  find  a  solution  for  the  above  set  of  equations  by  solving  a  linear  pro¬ 
gramming  problem,  as  the  following  lemma  states. 

Lemma  8.  To  determine  Prf  <f  for  all  s  £  S',  it  suffices  to  find  the  set  of  values 
{a;s  isGS'}  that  maximizes  y~].g  g,  xs  subject  to  the  set  of  constraints 

Xs  <  p\  {t)xt  +  pi  (■ t ) 

t£Sd 

for  all  s  6  S'p  and  I  <  *  <  ks.  Then,  it  is  simply  P r“  <f  =  xs,  for  all  s  6  S'p.  These 
values  are  well-defined,  as  the  above  problem  admits  a  unique  optimal  solution. 

To  solve  the  above  linear  programming  problem,  it  is  possible  to  use  well- 
known  algorithms,  such  as  the  simplex  method.  To  state  the  results  about  the 
complexity  of  pCTL  model  checking,  assume  that  II  is  described  by  listing  all 
the  next-state  distributions  for  all  states  as  vectors  of  rational  numbers,  each 
represented  as  the  ratio  of  two  integers.  The  size  of  II ,  denoted  by  \II\,  will  be 
simply  the  length  of  this  description,  considered  as  a  string.  Using  algorithms 
based  on  the  ellipsoid  method,  the  above  linear  programming  problem  can  be 
solved  in  polynomial  time  in  |T7|  [21].  Therefore,  we  have  the  following  theorem. 

Theorem  9.  If  the  truth-values  ofj,  if  are  knoivn  at  all  s  £  S,  the  truth-value 
of  lP<a {pf  II  if)  at  all  s  6  S  can  be  computed  in  polynomial  time  in  \II\. 

Computation  of  Pr^  <f>.  In  the  case  of  Pr+  <f,  the  set  S> o  =  {s  6  $  |  Pr+  <f  > 
0}  is  simply  the  set  of  states  of  the  directed  graph  ( Sd  U  Sp,p)  from  which  it 
is  possible  to  reach  Sd  following  a  path  belonging  to  the  graph  itself.  Again, 
Pr+  <f  =  0  for  s  6  S  —  S>o,  and  Pr^"  <f  =  1  for  S  6  Si-  Letting  S'p  d=f  5>o  —  Sd, 
for  all  s  6  S'p  we  can  write,  in  analogy  to  (11), 


Pr  U 


max 
1  <i<ks 


pi (*) 

i  fc,\i 


Again,  we  can  compute  Pr+  if  for  all  s  6  Sp  by  solving  a  linear  programming 
problem,  and  the  analogous  of  Theorem  9  holds  for  P>a<^. 


Complexity  of  pCTL  model  checking.  Combining  the  results  about,  the 
complexity  of  CTL  model  checking  [4]  with  Theorem  9,  we  get  the  following 
theorem  about  the  complexity  of  pCTL  model  checking  on  PNS. 

Theorem  10.  Model  checking  of  pCTL  formulas  over  a  PNS  IT  can  be  done  in 
time  polynomial  in  \II\  and  linear  in  the  size  of  the  formula. 


4.2  pCTL*  Formulas 

We  now  turn  to  the  problem  of  computing  Pr~  and  Pr+  f  for  a  general  pCTL* 
path  formula  <fi  6  Seq.  As  Pr“  <fi  =  1  —  Pr}"  -if  by  Lemma  6,  we  need  to  consider 
only  the  case  of  Pr}"  f.  As  usual,  we  assume  that  the  truth- values  of  all  state 
subformulas  of  <fi  have  already  been  evaluated  at  all  states  of  the  system. 

The  algorithm  we  propose  consists  of  three  steps.  First,  we  put  the  formula 
f  in  a  canonical  form  <j>" .  Second,  we  construct  from  II  a  new  system  II' ,  such 
that  the  states  of  W  keep  track  of  the  truth-values  of  the  subformulas  of  f , 
and  the  probability  of  sets  of  sequences  in  II  is  equal  to  the  probability  of  the 
corresponding  sets  of  sequences  in  II' .  Third,  we  show  that  computing  Pr}"  f  in 
II  corresponds  to  computing  the  probability  of  reaching  certain  sets  of  states  of 
II' ,  and  this  can  be  done  using  the  method  previously  outlined  for  pCTL.3 

Canonical  form  for  <f>.  Let  F  =  {71,...,%}  be  the  set  of  maximal  state 
subformulas  of  <j>,  that  is,  the  set  of  state  subformulas  of  f  that  are  not  proper 
subformulas  of  any  other  state  subformula  of  f .  For  each  7 i,  we  introduce  a 
new  propositional  variable  77,  and  let  <f>‘  =  <^[)’i/7i]  •  •  •[ Tn/ln]  be  the  result  of 
replacing  each  occurrence  of  7 i  in  f  with  77,  for  all  1  <  i  <  n.  As  for  each  state 
s  6  S  we  have  already  computed  whether  s  |_  7. .  we  can  extend  the  labeling  V 
by  letting  C(s)  =  C(s)  U  {77  |  s  |=  7;,  1  <  i  <  n}. 

The  resulting  formula  <f>'  is  a  linear-time  temporal  formula  constructed  with 
the  propositional  connectives  and  the  temporal  operators  □ ,  <>,  U  on  the  propo¬ 
sitional  variables  n, . . . ,  rn  [17].  By  the  results  of  [16,  3],  —if  can  be  put  into  the 
canonical  form  A/-  1  (nO/C  V  OO  A  ,■ )  for  some  past  temporal  formulas  \i ,  •  •  • ,  Xl  1 
Ai,  . . . ,  A;  built  with  propositional  connectives  and  the  past  temporal  operators 
S  (since)  and  ©  ( previous )  [15,  17].  Thus,  <$>'  can  be  put  into  the  form 

1 

<f"  :  \j  on  (A  A  o  ipi)  , 

i  —  1 

where  again  ...,  Si,  ij>i,  ...,  ifi  are  past  temporal  formulas.  Moreover,  the 
size  of  (j)"  is  at  most  doubly  exponential  in  the  size  of  <j>. 

Construction  of  II'.  The  truth-value  of  a  past  formula  at  point  s/.  of  a  se¬ 
quence  so,  si,  S2,  ■  ■  ■  depends  only  on  the  finite  “past”  sequence  so,si,...,sj;. 
Therefore,  it  is  possible  to  construct  from  II  =  (S,  Sin,V,  r)  a  system  II 1  = 
(S',  s'nJ  V ,  t')  whose  states  keep  track  of  the  truth-values  of  the  past  formulas 
in  (j>"  as  II  follows  a  sequence  of  states. 

To  do  so,  let  9i,  . . . ,  9m  be  the  set  of  past  subformulas  of  <f"  having  S  or 
©  as  the  main  connective,  ordered  in  such  a  way  that  no  A  is  a  subformula  of 
6j  for  i  >  j.  The  space  state  of  W  is  S'  =  S  x  {true, false}"1 ,  so  that  a  state 
s'  -  (s,  Ui, . . . ,  Um)  £  S'  consists  of  a  state  s  of  II  and  of  a  sequence  «!,...,  um 

3  An  alternative  approach,  not  pursued  in  this  paper,  would  have  been  to  construct 
77,  from  i7  and  from  a  deterministic  Street  automaton  for  -if. 


of  truth- values  of  9i ,  . .  . ,  9m .  Any  state  in  S'  can  be  taken  as  the  initial  state  «• 
of  II' .  We  define  the  projection  function  7r  :  S'  e- y  S  by  7r((s,  u%, . . . ,  um))  =  s. 
Let  q i,  . . . ,  qm  be  new  propositional  variables,  that  will  be  used  to  replace  the 
formulas  9\ ,  . .  . ,  9m.  The  labeling  function  l  '  is  defined  by 

Ml, . . . ,  Um))  —  V (s)  U  {qi  I  Ui  =  true,  1  <  i  <  m}  . 

For  1  <  i  <  m,  define  9{  =  (. . .  (9i\qi-i/6i-if)  . . .  [<Zi/#i])  to  be  the  formulas 
resulting  from  successively  substituting  in  9i  qi-i,  . . . ,  q  1  for  . . . ,  9 1.  For 

l  <  k  <  l,  define 

4  =  (•  •  •  (4 [5m/ftn])  •  •  •  [51/^1])  4>k  =  (■■■  ($k[qm/6m))  ■  ■  ■  [n/61]) 

to  be  the  propositional  formulas  resulting  from  successively  substituting  qm ,  . .  . , 
q  1  for  61,,, ,  . . . ,  9 1  in  (5^,  tpi..  Note  that  (9,-  does  not  contain  any  qj  for  1  <  i  <  j  < 
m.  Define  also  (j)  to  be  the  formula  obtained  by  orderly  replacing  each  £,• ,  1 /),■  in 
<fi"  with  h, ,  ijj, .  I  S  i  <  t.  respectively. 

The  definition  of  the  reachability  relation  p'  in  U'  encodes  the  semantics 
of  the  past  operators.  Recall  that  a  formula  a  S  f3  holds  at  a  given  state  of  a 
sequence  if  j3  holds  at  that  state,  or  if  a  holds  at  that  state  and  a  S  f3  holds  at 
the  previous  one;  a  formula  ©o-  holds  at  a  given  state  if  a  holds  at  the  previous 
one.  Consider  any  two  states  s'  =  (s,  u\ , . . . ,  um ),  t'  =  (t,v  1, .  . . ,  vm)  of  II' .  As 
Ui ,  V(  represent  the  truth-values  of  9i  at  s' ,  t'  respectively,  we  let  t'  be  reachable 
from  s',  written  p'(s',t'),  if  p(s,t)  and,  for  all  I  <  i  <  m: 

1.  if  9i  has  the  form  ©a,  V{  =  true  iff  s'  |=  cv; 

2.  if  9{  has  the  form  a  S  /?,  Vi  =  true  iff  [ t '  \ —  p  or  (m7-  =  true  and  t'  |=  o-)] . 

The  next-state  probability  distributions  for  II'  are  then  defined,  for  s'  £  S' , 
ksi  =  kw(si~),  and  for  1  <  i  <  ksi,  by: 

pftt')  =  | Pi[s )(7r(i'))  if 

f  0  otherwise. 

The  fact  that  the  above  equation  defines  next-state  probability  distributions  is 
a  consequence  of  the  following  lemma. 

Lemma  11.  Given  s  6  S  and  s'  £  S'  such  that  s  =  for  every  t  6  S  such 

that  p{s,t)  there  is  exactly  one  t'  £  S'  such  that  t  =  71 ft')  and  p'{s',t'). 

Proof.  Let  t'  =  (r,v  1, . . . ,  vm)  be  a  state  in  S'  such  that  t  =  7r{t')  and  p' (s' ,  t'). 
The  value  of  r  is  uniquely  determined  by  r  =  t.  For  1  <  i  <  rri,  the  truth  value 
of  Vi  is  determined  by  s' ,  t  and  by  the  truth  values  of  r  - .  . .  . ,  Vi- 1.  Hence,  t'  is 
uniquely  determined.  □ 


Relationship  between  77  and  U' .  A  formula  ©cv  is  always  false  on  the  first 
state  of  a  sequence.  A  formula  a  S  j3  holds  on  the  first  state  of  a  sequence  if  that 
state  satisfies  j3.  Thus,  in  order  for  u\,...,um  to  represent  the  truth- value  of 

(>■ . 0,ri ,  a  sequence  in  17  that  starts  at  the  state  s£S  should  start  in  U'  at  a 

state  ((s)  =  (s,  u.\, . . . ,  um)  such  that,  for  all  I  <  i  <  m,  u,-  is  true  iff  0,-  has  the 
form  a  S  (3  and  ((s)  |=  /?.  As  the  above  requirement  uniquely  determines  ((s), 
it  defines  a  one-to-one  function  (  :  S  S' , 

Moreover,  for  all  s  £  S,  there  is  a  bijective  correspondence  between  the  legal 
sequences  of  II  that  start  at  s  6  S'  and  those  of  U'  that  start  at  ((s).  This 
correspondence  relates  each  legal  sequence  to  :  .v(!.  .  Sy. ■  ■  ■  of  II  with  the  unique 

legal  sequence  ((_)  :  ,s(, ,  s'- ,  s'2 . . . .  of  II'  such  that  ((so)  =  So;  an<^  "(•*>';)  = 
for  all  i  >  0.  If  A  6  Qs  is  a  set  of  sequences  of  17,  denote  with  ((Z\)  the  set  of 
(■-related  sequences  in  II'.  The  following  lemma  follows  from  the  construction  of 
II'  and  <j>. 

Lemma  12.  w  \=  <f>  iff  ((o>)  | =  (f>,  so  that 

(({w  G  I2S  |  w  1=  <j>})  =  {ui'  G  \uj'  \=cp}  . 

Proof.  Given  two  corresponding  sequences  ui  :  so,  si,  S2,  ■  . .,  ((w)  :  /l:,  / • .  /•_>. . . . 
with  labelings  V,  V'  respectively,  p  holds  at  s o  iff  4>"  holds  at  sq.  By  induction 
on  i  it  can  be  proved  that  9i  holds  at  s &  iff  holds  at  tp,  iff  «,■  =  true  at  tp, ,  for 
1  <  i  <  m,  k  >  0.  Hence  <fi"  holds  at  s o  iff  holds  at  to,  and  this  concludes  the 
proof.  □ 

Furthermore,  there  is  a  correspondence  between  the  strategies  of  17  and  II' . 
To  i)  for  17  corresponds  rj  for  II'  such  that 

Qp{i  I  4  •  •  -4)  ~  Qv(*  I  ^(s'o)  •  •  -^(4))  ,  (!2) 

for  all  n  >  0,  all  sequences  s'Q  .  .  ,s'n  of  states  of  II',  and  I  <  i  <  ksi.  Related  sets 
of  sequences  starting  from  related  states  of  17,  II'  have  thus  the  same  probability, 
as  the  following  lemma  states. 

Lemma  13.  If  A  6  Bs  is  a  measurable  set  and  r),  i)'  are  related  as  in  (12), 
fis  v(A)  =  /i£(a)  ^/(((Z\)).  Therefore,  by  definition  of  maximal  measure, 

l*f(A)  =  ■ 

Proof.  The  result  follows  easily  from  the  definition  of  next  step  probabilities  in 
II'  and  from  the  fact  that  (  is  one-to-one  and  (  is  bijective.  □ 


Computing  in  II' .  From  the  above  relations,  in  order  to  compute  Pr+  <f>  in  17 
it  suffices  to  compute  Pr^s)  4>  in  17';  and  to  compute  this  we  can  take  advantage 

of  the  special  form  of  <j)  :  V!=i  On(<5j  A  o4)- 

For  1  <  i  <  /,  define  Cj  =  {s  6  S'  \  s  |=  <5,},  set  B{  :=  Cj,  and  iterate  the 
following  three-step  procedure  until  no  more  states  can  be  removed  from  7( . 


1.  Define,  for  each  s  £  B;,  the  set  of  indices 

Ms  =  [j  e  {1, . . . , ks]  |  {t  e  s'  I  pj(t)  >  0}  c  b*} 

of  next-state  distributions  that  do  not  lead  any  computation  outside  B,- . 

2.  Consider  the  directed  graph  G  =  (B,-,  E),  where 

E  =  {(«./)  I  3 j  £  Ms  >  0}  . 

3.  Remove  from  B,-  all  states  s  that  cannot  reach  a  state  in  {s  £  B,-  |  s  |=  f>i} 
by  a  path  in  G  of  length  at  least  1 . 

Note  that  the  above  procedure  is  iterated  TV,-  <  IS7]  times. 

For  1  <  *  <  /,  let  Fi  be  the  subsets  of  B(  obtained,  and  let  F  =  UL  Fi.  For 
A  C  S' ,  s  £  S' ,  define  rs(A)  =  {w  £  £2's  |  3&.w|*  6  A}  to  be  the  set  of  sequences 
that  reach  A  from  s.  The  following  theorem  allows  us  to  compute  Pr^"  (j). 

Theorem  14.  For  s  £  S,  Pr f  cp  =  j 

The  quantity  P^s)(F((s)(F))  can  then  be  computed  with  the  algorithm  given 
in  the  previous  section  for  pCTL,  taking  F  as  Sd  and  S'  as  Sp.  The  proof  of  the 
theorem  uses  the  two  following  lemmas. 

Lemma  15.  For  all  s  6  S' ,  there  is  a  strategy  r)  such  that  a  sequence  ui  6  rs(F) 
satisfies  (j)  ivith  probability  1,  i.e. 

pUrs(F))  =pU{u  ers(F)  \  w  M})  • 

Moreover,  this  strategy  does  not  need  to  depend  on  the  portion  of  to  6  rs(F) 
outside  F . 

Proof.  Assume  that  to  6  F  is  the  first  state  at  which  ui  6  rs{F)  enters  F.  Let 
i  =  min  y  jn  |  1  <  m  <  l  A  to  6  Ftn  } .  For  t  6  Fi,  let 

Mt  =  [, j  G  {1,  •  ..,kt}  {t'  e  S'  |  p){t')  >  0}  C  B,} 

be  the  set  of  indices  of  next-state  distributions  that  do  not  leave  Fi.  The  strategy 
y,  at  t  6  Fi,  will  choose  one  of  the  j  £  Mt  with  equal  probabilities.  Note  that 
while  the  strategy  depends  on  the  state  t0  of  first  entry  in  F ,  it  does  not  depend 
on  the  portion  of  to  outside  F .  After  the  entry  in  F ,  the  sequence  is  confined  to 

Fi ;  from  each  t  £  Fi  there  is  a  path  to  a  state  of  B,-  where  ipi  holds;  and  B,-  has 

finite  size.  Therefore,  the  sequence  io  will  satisfy  <>(n^i  A  □OV’i);  and  <fi,  with 
probability  1 .  □ 

Lemma  16.  For  1  <  i  <  l,  s  £  S',  and  for  any  strategy  rj,  the  measure  of  the 
set  of  sequences  from  s  that  satisfy  A  □OV'i)  without  ever  entering  Fi  is 

0,  i.e. 

Ps,v  ({w  £f2's\w\=  <*>(□£  A  DOA)}  ~  BS(B,))  =  0  . 


Proof.  For  1  <  j  <  N; .  let  Dj  be  the  set  of  states  that  have  been  removed  from 
Bi  at  the  |-th  iteration  of  the  procedure;  let  also  Dq  -  S'  —  C;  be  the  set  of  states 
that  does  not  satisfy  $,-.  Let  l)<j  =  Dj,  D>j  =  .+1  Dj.  Moreover,  call 

a  ^-state  any  state  i  fc  S'  such  that  t  |=  ifi .  Define  also 

6  =  inf {psm  (/)  |  S;t  e  S'  A  1  <  m  <  ks  A  psm  (t)  >  0} 

and  note  that  b  >  0,  as  S'  has  finite  size.  We  will  prove  the  following  assertion 
by  complete  induction  on  j,  from  TV,-  down  to  1: 

For  1  <  j  <  TV,’,  a  sequence  passing  from  s  6  Dj,  never  entering  Fi  and 
satisfying  □OV'i  will  contain  a  state  in  D<j  ivith  probability  1. 

Clearly,  this  assertion  implies  the  result  stated  by  the  lemma. 

Consider  the  case  of  j,  1  <  j  <  TV,-,  and  assume  that  the  assertion  has  been 
proved  for  all  j' ,  j  <  j'  <  TV,-. 

Let  a%  be  the  fraction  of  sequences  passing  through  s  6  Dj  and  reaching  a 
i/j,- -state  without  leaving  Dj  U  D>  j .  Since  s  has  been  removed  from  Bi ,  each  of 
these  sequences,  before  reaching  the  i^-state,  must  pass  through  a  critical  point , 
i.e.  a  point  where  the  strategy  ij  has  chosen  a  next-state  probability  distribution 
p  such  that  {t  6  S'  \  p(t)  >  0}  $2  Dj  U  D>  j.  Therefore,  ai  <  1  —  b,  as  at  most 
1  —  6  sequences  that  pass  through  a  critical  point  remain  in  Dj  U  D>  j . 

The  ipi -state  reached  by  the  a i  sequences  belongs  to  either  Dj  or  D>  j .  If 
it  belongs  to  Dj,  we  say  that  the  first  cycle  is  concluded.  Otherwise,  by  the 
induction  hypothesis  we  know  that  the  sequences  that  pass  through  D>j  and 
satisfy  □OV'i  without  entering  Fi  eventually  go  to  a  state  in  Dj  U  D<j  with 
probability  1.  For  these  sequences,  the  first  cycle  is  concluded  when  they  reach 
Dj  U  D<j.  In  either  case,  at  most  Gp  sequences  complete  the  first  cycle  without 
leaving  Dj  or  D>j. 

A  fraction  aj  of  the  sequences  that  complete  the  first  cycle  without  leaving 
DjUDyj  will  reach  another  ifi -state  without  leaving  DjUDyj .  As  they  must  pass 
again  through  a  critical  point,  a2  <  1—6.  In  general,  the  fraction  of  sequences  that 
goes  through  k  cycles  without  leaving  Dj  U  D>j  is  at  most  rim.=i  am  <(1-6)*. 
Therefore,  the  set  of  sequences  passing  through  s  6  Dj  that  satisfy  □<>  Vl’  without 
leaving  Dj  U  D>j  has  measure  0.  □ 

Corollary  17.  For  any  s  6  S'  and  rp  ps 6  D's  \  uj  |=  <j>}  —  rs(F ))  =  0. 

Proof.  From  Lemma  16  we  have 

Vs,v({u  £  f2's  \  ui  \=  (j>}  -  rs{F)) 

l 

<  e  n's  I  w  1=  A  □OV’i)}  -  Ps(Fi))  =  o  . 

i  —  1 


□ 


Proof  of  Theorem  If.  For  s'  £  S',  by  the  definition  of  maximal  probability  we 
have  Pr+</>  =  supr(  fisi  ,v({uj  £  [2't,  \  u  |=  </>}).  By  Corollary  17  we  have,  for  any 
strategy  r), 

e  c',  |  w  |=  if}) 

=  Us1, r)  (V  6  rs'{F)  I  ui  |=  </>})  +  ({w  £  C2's I  \  UJ  \=  (f)}  -  rsi(F)) 

=  Vs',v  (V  £  Fs ,  (F)  |  w  M})  . 

Hence,  by  Lemma  15,  Pr  +,</>  =  sup^  psi:V({u  £  Fsi(F)  \  ui  |=  <j>})  —  pf,(rsi(F)). 
From  Lemmas  12  and  13  we  finally  have  Pr+ <j>  =  Pr^s^  =  A4^)  (/f(s)(i?)) ,  as 
was  to  be  proved.  □ 


Complexity  of  pCTL*  model  checking.  By  combining  results  about  the 
complexity  of  CTL*  model  checking  [7],  pCTL  model  checking,  and  an  analysis  of 
the  above  algorithm,  we  get  the  following  result,  that  summarizes  the  complexity 
of  pCTL*  model  checking  for  PNS. 

Theorem  18.  Model  checking  of  pCTL*  formulas  over  a  PNS  II  can  be  done 
in  polynomial  time  in  \II\. 

On  the  other  hand,  from  the  results  of  [5]  we  know  that  determining  whether 
a  linear-time  temporal  formula  is  satisfied  with  probability  1  by  a  PNS  requires 
at  least  doubly  exponential  time  in  the  size  of  the  formula.  As  this  problem  can 
be  reduced  to  pCTL*  model  checking,  we  have  the  following  result. 

Theorem  19.  Model  checking  of  pCTL*  formulas  over  PNS  has  a  time  com¬ 
plexity  that  is  at  least  doubly  exponential  in  the  size  of  the  formula. 

In  the  algorithm  we  presented,  we  can  trace  the  source  of  this  complexity  to 
the  step  that  computes  the  canonical  form  of  a  temporal  formula,  and  to  the 
construction  of  LI' .  In  fact,  \n'\  is  triply  exponent ial  in  \<j>\,  in  the  worst  case. 


Strategies  for  pCTL  and  pCTL*.  We  say  that  a  strategy  rj  is  deterministic 
if  Qv(i  |  s0  ■  ■  .sn)  is  either  0  or  1  for  all  I  <  i  <  kSii,  n  >  0  and  all  sequences 
s0  .  .  .sn  of  states  of  S.  We  say  that  a  strategy  is  Markovian  if 

Qy  (1  |  i  I  ^ n  ) 

for  all  n  >  0  and  all  sequences  s0  ■ . .  sn  of  states  of  S. 

Given  a  system  II ,  and  <f>  £  Seq,  s  £  S,  say  that  a  strategy  rj  is  most  favorable 
(resp.  most  unfavorable)  if  ps  r)({uj  £  I2S  \  ui  |=  </>})  =  Prj" <f>  (resp.  if  ps  ri({w  £ 
f2s  |  w  |=  f})  =  Pr“</>).  The  following  corollary,  derived  from  an  analysis  of  the 
model-checking  algorithms,  gives  us  a  characterization  of  the  most  favorable  and 
unfavorable  strategies  corresponding  to  pCTL  and  pCTL*  formulas. 


Corollary  20.  The  following  results  hold. 


1.  For  all  PNS  II  and  all  pCTL  formulas  <j>  £  Seq,  there  are  Markovian  and 
deterministic  strategies  that  are  most  favorable  and  most  unfavorable  for  all 
s  ES. 

2.  For  all  PNS  II,  all  pCTL*  formulas  <fi  6  Seq  and  all  s  6  S,  there  are  most 
favorable  and  most  unfavorable  strategies  that  are  deterministic.  However, 
there  are  PNS  II ,  s  6  S,  and  pCTL*  formulas  6  Seq  such  that  there  are 
no  most  favorable  nor  most  unfavorable  strategies  that  are  Markovian. 

The  second  part  of  this  corollary  shows  that  nondeterminism  cannot  be  en¬ 
coded  by  leaving  some  transition  probabilities  of  a  Markov  chain  unspecified,  if 
pCTL*  is  used  as  the  specification  language. 

5  Conclusions 

It  is  known  from  [10,  2]  that  pCTL  and  pCTL*  model  checking  on  Markov  chains 
can  be  done  in  polynomial  time  in  the  size  of  the  system.  It  is  interesting  to  note 
that  adding  nondeterminism  still  preserves  the  polynomial  time  bound,  provided 
the  size  of  the  system  takes  into  account  not  only  the  number  of  states,  but  also 
the  encoding  of  the  transition  probabilities. 

The  situation  is  different  for  the  time  bounds  expressed  in  terms  of  the  size 
of  the  formula.  Model  checking  of  pCTL  formulas  can  be  done  in  linear  time  on 
the  size  of  the  formula  both  for  Markov  chains  [10]  and  PNS.  However,  while 
pCTL*  model  checking  on  Markov  chains  can  be  done  in  single  exponential  time 
in  the  size  of  the  formula  [5,  2],  pCTL*  model  checking  on  PNS  requires  at 
least  doubly  exponential  time  in  the  size  of  the  formula.  In  our  algorithm,  the 
complexity  of  putting  formulas  in  canonical  form  is  partially  mitigated  by  the 
fact  that  many  common  formulas  used  in  system  specification  can  be  efficiently 
put  into  canonical  form. 
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